[CALCITE-7779] Optimization rule FilterAggregateTransposeRule rewrites queries to semantically non-equivalent ones - #5260
Conversation
… column indices without remapping for non-simple aggregates
|
Closing because the commit title is not the same as the Jira ticket title |
|
|
you know, git has a command to rewrite a commit |
|
I figured from Calcite's Developer Guide that it would be preferable to open a new PR. In order to update the pull request, you need to commit the changes in your branch and then push the commit(s) to GitHub. You are encouraged to use regular (non-rebased) commits on top of previously existing ones. When pushing the changes to GitHub, you should refrain from using the --force parameter and its alternatives. If that was the wrong choice, I'll keep that in mind in the future. |
|
You can do whatever you want in your fork, just don't do it in the upstream calcite repository. |
|
Thanks for the tips |



Jira Link
CALCITE-7779
Filter pushability
In the simple aggregate case, FilterAggregateTransposeRule pushes a predicate below an aggregate when the predicate's referenced columns all belong to the aggregate’s grouping set, because aggregate-function results are unavailable before aggregation.
In the non-simple aggregate case, FilterAggregateTransposeRule pushes a predicate below an aggregate when the predicate's referenced columns all belong to every grouping set, because a referenced column is replaced with NULL in rows produced by a grouping set that omits it, and evaluating the predicate before aggregation can change the result.
Bug description
The pushability check in the non-simple case currently compares filter-input column positions (= aggregate-output column positions) with aggregate-input column positions without remapping. The same position can identify different columns in these two row layouts, so this check does not establish that the predicate’s referenced columns belong to every grouping set. Consequently, the rule can incorrectly push a predicate and change the query result, or fail to push a predicate when doing so would be valid.
Reproducer
After applying AGGREGATE_PROJECT_MERGE followed by FILTER_AGGREGATE_TRANSPOSE, the resulting plan is equivalent to:
which isn't semantically equivalent to the first query.
Fix
The fix uses
Mappings.targetandImmutableBitSet.permuteto express each grouping set in aggregate-output positions before comparing it with the predicate's references. The existing simple-aggregate check is unchanged.## Tests
The existing dedicated grouping-set tests use matching input and output positions. The new tests apply
AGGREGATE_PROJECT_MERGEfirst to expose the mismatch and cover:HAVING b IS NULL: keep the filter above the aggregate to preserve the subtotal row.HAVING b = 2: keep the filter above the aggregate to avoid an extra subtotal row.HAVING a = 1: allow pushdown becauseabelongs to every grouping set.All three regression tests failed against the original rule.